Papers with artificial agents
A Benchmark for Recipe Understanding in Artificial Agents (2024.lrec-main)
Copied to clipboard
Jens Nevens, Robin de Haes, Rachel Ringe, Mihai Pomarlan, Robert Porzel, Katrien Beuls, Paul van Eecke
| Challenge: | a benchmark has been designed to evaluate whether artificial agents are able to understand how to perform everyday activities. |
| Approach: | They propose a benchmark task that maps a recipe to a set of cooking actions that are precise enough to be executed in the simulated kitchen. |
| Outcome: | The proposed benchmark consists of mapping a recipe to a set of cooking actions that is precise enough to be executed in the simulated kitchen and yields the desired dish. |
Grounding Meaning Representation for Situated Reasoning (2022.aacl-tutorials)
Copied to clipboard
| Challenge: | a tutorial aims to build agents that understand language using a simulated environment . situated reasoning is a critical aspect of human language understanding . |
| Approach: | This tutorial combines a synthesis of multimodal grounding and meaning representation techniques with formal and computational models of situated reasoning. |
| Outcome: | This tutorial combines multimodal grounding and meaning representation techniques with formal and computational models of embodied reasoning. |
Seeded self-play for language learning (D19-64)
Copied to clipboard
| Challenge: | Current methods for learning human language are too data inefficient to learn it in this way. |
| Approach: | They propose to train a meta-learning agent in simulation to interact with populations of pre-trained agents, each with their own distinct communication protocol. |
| Outcome: | The proposed algorithm minimizes the number of on-policy interactions while learning human language while minimizing the number on-political interactions. |
What Action Causes This? Towards Naive Physical Action-Effect Prediction (P18-1)
Copied to clipboard
| Challenge: | a new task on naive physical action-effect prediction addresses the relationship between concrete actions and their effects on the state of the physical world as depicted by images. |
| Approach: | They propose a task that harnesses web image data to facilitate action-effect prediction. |
| Outcome: | The proposed approach harnesses web image data through distant supervision to facilitate learning for action-effect prediction. |
Generalization in Text-based Games via Hierarchical Reinforcement Learning (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Reinforcement Learning (RL) based agents are promising for text-based games, but their generalization remains a challenge. |
| Approach: | They propose a hierarchical framework for reinforcement learning based on knowledge graphs . they propose to decompose the game into subtasks and execute a sub-policy in the low level to conduct goal-conditioned reinforcement learning. |
| Outcome: | The proposed framework enjoys favorable generalizability on a set of difficulty levels and is able to handle complex training tasks. |
Language-based General Action Template for Reinforcement Learning Agents (2021.findings-acl)
Copied to clipboard
| Challenge: | Prior knowledge is important in decision-making, and humans preserve it in the form of natural language (NL). |
| Approach: | They propose an environmentagnostic action framework that incorporates prior knowledge into decision-making . they propose to use general semantic schemes to facilitate agent in finding plausible actions . |
| Outcome: | The proposed agent performs better than agents that rely on gamespecific actions. |
Disentangling Categorization in Multi-agent Emergent Communication (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent work on the emergence of language between artificial agents has not isolated the effect of categorization power on inter-communication ability. |
| Approach: | They propose to use disentangled representations to quantify categorization power of agents to enable differential analysis between combinations of heterogeneous systems. |
| Outcome: | The proposed method reduces signaling accuracy by 40% despite encouraging compositionality in the artificial language. |
Internal and external pressures on language emergence: least effort, object constancy and frequency (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies show that the emergent languages rarely display salient features inherent to natural languages, such as compositionality of meaning and generalisation to novel objects. |
| Approach: | They propose to formalise the principle of least effort through an auxiliary objective and explore several game variants inspired by the principle 'object constancy' they find that the proposed sources of pressure result in emerging languages with less redundancy, more focus on high-level conceptual information, and better abilities of generalisation. |
| Outcome: | The proposed sources of pressure result in emerging languages with less redundancy, more focus on high-level conceptual information, and better abilities of generalisation. |
Multimodal Contextualized Semantic Parsing from Speech (2024.acl-long)
Copied to clipboard
| Challenge: | Towards this goal, we introduce Semantic Parsing in Contextual Environments (SPICE) task designed to enhance artificial agents’ contextual awareness by integrating multimodal inputs with prior contexts. |
| Approach: | They introduce a task designed to enhance artificial agents’ contextual awareness by integrating multimodal inputs with prior contexts. |
| Outcome: | The proposed task is based on the VG-SPICE dataset and the Audio-Vision Dialogue Scene Parser (AViD-SP) it allows agents to maintain their contextual state within a structured, dense information framework that is scalable and interpretable . |
Emergence of Hierarchical Reference Systems in Multi-agent Communication (2022.coling-1)
Copied to clipboard
| Challenge: | a hierarchical reference system allows the selection of the most appropriate level of specificity for a given context. |
| Approach: | They propose a hierarchical reference game to study the emergence of hierarchic reference systems in artificial agents. |
| Outcome: | The proposed game shows that agents can generalize to new concepts . the hierarchical reference game is based on a simplified world . |
Towards a Conversation-Analytic Taxonomy of Speech Overlap (L18-1)
Copied to clipboard
| Challenge: | a taxonomy for classifying speech overlap in natural language dialogue is presented . the scheme classifies overlap on the basis of several features, including onset point, local dialogue history, and management behavior. |
| Approach: | They propose a taxonomy for classifying speech overlap in natural language dialogue . they describe the various dimensions of the scheme and show how it was applied to a corpus of collaborative dialogue based on onset point, dialogue history, and management behavior . |
| Outcome: | The proposed taxonomy classifies overlap on the basis of onset point, dialogue history, management behavior. |
A Picture is Worth a Thousand Words: Language Models Plan from Pixels (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent work uses pre-trained language models to reason about plans from text instructions in embodied visual environments. |
| Approach: | They propose to use pre-trained language models to reason about plan sequences from text instructions in embodied visual environments. |
| Outcome: | The proposed approach outperforms previous approaches on the ALFWorld and VirtualHome benchmarks. |
The Emergence of Compositional Languages in Multi-entity Referential Games: from Image to Graph Representations (2024.emnlp-main)
Copied to clipboard
| Challenge: | Language Emergence research uses jointly trained artificial agents to solve a task. |
| Approach: | They propose a multi-entity game in which targets include multiple entities that are spatially related. |
| Outcome: | The proposed multi-entity game shows that the emergent languages exhibit a considerable degree of compositionality, but not over all features. |